An Association-Based Method for Automatic Indexing with a Controlled Vocabulary

In this paper we describe and test a two stage algorithm based on a lexical collocation technique which maps from the lexical clues contained in a document representation into a controlled vocabulary list of subject headings. Using a collection of 4,626 INSPEC documents, we create a “dictionary” of associations between the lexical items contained in the titles, authors and abstracts and controlled vocabulary subject headings assigned to those records by human indexers using a likelihood ratio statistic as the measure of association. In the deployment stage, we use the dictionary to predict which of the controlled vocabulary subject headings best describe new documents when they are presented to the system. Our evaluation of this algorithm, in which we compare the automatically assigned subject headings to the subject headings assigned to the test documents by human catalogers, shows that we can obtain results comparable to and consistent with human cataloging. In effect, we have cast this as a classic partial match information retrieval problem. We consider the problem to be one of “retrieving” (or assigning) the most probably “relevant” (or correct) controlled vocabulary subject headings to a document based on the clues contained in that document. ∗To whom all correspondence should be addressed